Video generation requires synthesizing consistent and persistent frames with dynamic content over time. This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs). First, towards composing adjacent frames, we show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, brings a smooth frame transition without compromising the per-frame quality. Second, by incorporating the temporal shift module (TSM), originally designed for video understanding, into the discriminator, we manage to advance the generator in synthesizing more consistent dynamics. Third, we develop a novel B-Spline based motion representation to ensure temporal smoothness to achieve infinite-length video generation. It can go beyond the frame number used in training. A low-rank temporal modulation is also proposed to alleviate repeating contents for long video generation. We evaluate our approach on various datasets and show substantial improvements over video generation baselines. Code and models will be publicly available at https://genforce.github.io/StyleSV.
translated by 谷歌翻译
The deep learning community has witnessed an exponentially growing interest in self-supervised learning (SSL). However, it still remains unexplored how to build a framework for learning useful representations of raw music waveforms in a self-supervised manner. In this work, we design Music2Vec, a framework exploring different SSL algorithmic components and tricks for music audio recordings. Our model achieves comparable results to the state-of-the-art (SOTA) music SSL model Jukebox, despite being significantly smaller with less than 2% of parameters of the latter. The model will be released on Huggingface(Please refer to: https://huggingface.co/m-a-p/music2vec-v1)
translated by 谷歌翻译
近年来，由于其在数字人物，角色产生和动画中的广泛应用，人们对3D人脸建模的兴趣越来越大。现有方法压倒性地强调了对面部的外部形状，质地和皮肤特性建模，而忽略了内部骨骼结构和外观之间的固有相关性。在本文中，我们使用学习的参数面部发电机提出了雕塑家，具有骨骼一致性的3D面部创作，旨在通过混合参数形态表示轻松地创建解剖上正确和视觉上令人信服的面部模型。雕塑家的核心是露西（Lucy），这是与整形外科医生合作的第一个大型形状面部脸部数据集。我们的Lucy数据集以最古老的人类祖先之一的化石命名，其中包含正牙手术前后全人头的高质量计算机断层扫描（CT）扫描，这对于评估手术结果至关重要。露西（Lucy）由144次扫描，分别对72名受试者（31名男性和41名女性）组成，其中每个受试者进行了两次CT扫描，并在恐惧后手术中进行了两次CT扫描。根据我们的Lucy数据集，我们学习了一个新颖的骨骼一致的参数面部发电机雕塑家，它可以创建独特而细微的面部特征，以帮助定义角色，同时保持生理声音。我们的雕塑家通过将3D脸的描绘成形状混合形状，姿势混合形状和面部表达混合形状，共同在统一数据驱动的框架下共同建模头骨，面部几何形状和面部外观。与现有方法相比，雕塑家在面部生成任务中保留了解剖学正确性和视觉现实主义。最后，我们展示了雕塑家在以前看不见的各种花式应用中的鲁棒性和有效性。
translated by 谷歌翻译
尽管利用张量低级别先验的方法是在高维数据处理中蓬勃发展并获得了令人满意的性能，但它们在动态磁共振（MR）图像重建中的应用受到限制。在本文中，我们集中于基于快速傅立叶变换（FFT）的张量奇异值分解（T-SVD），并且仅提供了FFT域中的确定且有限的张量低级别先验密切的数据和FFT域匹配。通过将FFT推广到转换的T-SVD的任意统一转换并提出了转换的张量核标准（TTNN），我们引入了一个基于TTNN的灵活模型，能够利用张量的低量量，在变换的域中的张量低级别。更大的转换空间并精心设计了基于乘数交替方向方法（ADMM）的迭代优化算法，该算法进一步将其进一步展开为基于模型的深层展开的重建网络，以学习转换后的张量低率之前（t $^2） $ LR-NET）。卷积神经网络（CNN）被合并到T $^2 $ LR-NET中，以从动态MR Image数据集中学习最匹配的转换。展开的重建网络还通过利用CNN提取的特征域中的低级别先验来提供有关低级先验利用率的新观点。两个心脏CINE MR数据集的实验结果表明，与基于最新优化和基于网络的最先进的基于网络的方法相比，提出的框架可以提供改进的恢复结果。
translated by 谷歌翻译
弱监督指定的实体识别方法训练标签模型，以汇总多个嘈杂标签功能（LFS）的代币注释，而无需看到任何手动注释的标签。为了正常工作，标签模型需要在上下文上识别和强调表现出色的LF，同时降低表现不佳的情况。但是，由于缺乏地面真理，评估LFS是具有挑战性的。为了解决这个问题，我们提出了稀疏条件隐藏的马尔可夫模型（稀疏-CHMM）。稀疏-CHMM并没有将整个发射矩阵视为其他基于HMM的方法，而是专注于估计其对角线元素，这些元素被认为是LFS的可靠性得分。然后将稀疏分数扩展到具有预定义膨胀函数的全面发射矩阵。我们还通过加权XOR分数来增强发射，该分数跟踪LF观察不正确实体的概率。通过三阶段的训练管道通过无监督的学习来优化稀疏-CHMM，从而降低了训练难度并防止模型落入本地Optima。与扳手基准中的基线相比，稀疏-CHMM在五个综合数据集上取得了3.01的平均F1分数提高。实验表明，稀疏-CHMM的每个组件都是有效的，估计的LF可靠性与真实LF F1分数密切相关。
translated by 谷歌翻译
这项工作旨在将在一个图像域上预先训练的生成的对抗网络（GaN）转移到新域名，其仅仅是只有一个目标图像。主要挑战是，在有限的监督下，综合照片现实和高度多样化的图像非常困难，同时获取目标的代表性。不同于采用Vanilla微调策略的现有方法，我们分别将两个轻量级模块导入发电机和鉴别器。具体地，我们将属性适配器引入发电机中冻结其原始参数，通过该参数，它可以通过其重复利用现有知识，因此保持合成质量和多样性。然后，我们用一个属性分类器装备了学习良好的鉴别器骨干，以确保生成器从引用中捕获相应的字符。此外，考虑到培训数据的多样性差（即，只有一个图像），我们建议在培训过程中建议在生成域中的多样性限制，减轻优化难度。我们的方法在各种环境下提出了吸引力的结果，基本上超越了最先进的替代方案，特别是在合成多样性方面。明显的是，我们的方法即使具有大域间隙，并且在几分钟内为每个实验提供鲁棒地收敛。
translated by 谷歌翻译
预训练为深入学习支持的X线射线分析中最近的成功奠定了基础。它通过在源域上进行大规模完全监督或自我监督的学习来学习可转移的图像表示。然而，监督的预培训需要复杂和劳动密集的两级人类辅助注释过程，而自我监督的学习不能与监督范例竞争。为了解决这些问题，我们提出了一个跨监督的方法，命名为审查监督（指的）的自由文本报告，该报告从射线照相中获取来自原始放射学报告的自由监督信号。该方法采用了视觉变压器，旨在从每个患者研究中的多种视图中学习联合表示。在极其有限的监督下，引用其在4个众所周知的X射线数据集上的转移学习和自我监督学习对应。此外，甚至是基于具有人辅助结构标签的射线照相的源区的甚至超越方法。因此，有可能取代规范的预训练方法。
translated by 谷歌翻译
translated by 谷歌翻译
Existing 3D-aware image synthesis approaches mainly focus on generating a single canonical object and show limited capacity in composing a complex scene containing a variety of objects. This work presents DisCoScene: a 3Daware generative model for high-quality and controllable scene synthesis. The key ingredient of our method is a very abstract object-level representation (i.e., 3D bounding boxes without semantic annotation) as the scene layout prior, which is simple to obtain, general to describe various scene contents, and yet informative to disentangle objects and background. Moreover, it serves as an intuitive user control for scene editing. Based on such a prior, the proposed model spatially disentangles the whole scene into object-centric generative radiance fields by learning on only 2D images with the global-local discrimination. Our model obtains the generation fidelity and editing flexibility of individual objects while being able to efficiently compose objects and the background into a complete scene. We demonstrate state-of-the-art performance on many scene datasets, including the challenging Waymo outdoor dataset. Project page: https://snap-research.github.io/discoscene/
translated by 谷歌翻译
Generative adversarial network (GAN) is formulated as a two-player game between a generator (G) and a discriminator (D), where D is asked to differentiate whether an image comes from real data or is produced by G. Under such a formulation, D plays as the rule maker and hence tends to dominate the competition. Towards a fairer game in GANs, we propose a new paradigm for adversarial training, which makes G assign a task to D as well. Specifically, given an image, we expect D to extract representative features that can be adequately decoded by G to reconstruct the input. That way, instead of learning freely, D is urged to align with the view of G for domain classification. Experimental results on various datasets demonstrate the substantial superiority of our approach over the baselines. For instance, we improve the FID of StyleGAN2 from 4.30 to 2.55 on LSUN Bedroom and from 4.04 to 2.82 on LSUN Church. We believe that the pioneering attempt present in this work could inspire the community with better designed generator-leading tasks for GAN improvement.
translated by 谷歌翻译